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ABSTRACT • . 

This speech^ con^cerns the diiiiensicns of . <^ - 
context- oriented e valuatipn ' an d begins nitfr a discussion cf the 
differences • between measurement, research, statistical methoddlogy, 
and evalua'tion. Decision-oriented evaluation is described as a 
process in- which the evaluator has, as a niajor concern, ^^e 
responsibility to' deter mine' likely decisicrs prioz to data collection 
or analysis.^ The assumptioft that t ha linkHb^weeri evalual^n and 
deci^ion-makilfq is direct and immediate, acd^that this as^proach is 
appropriate^ for. all si^uatidns, i^ seen as a drawback which.flimits ^ 
the decisicn-ori^nted evaluation approach, O^n describing the 
contextx oriented evaluation approach,^ the author provides a three 
dimensional metfiod for ^classifying -the evaluation context. The first 
dimensdbri concerns evaluation intent aid entails process, , results 
with a predisposi^tion, and openminded res.ults. The second dimension, 
of evaluation context iirvolves whether evaluations are ccffflissioned 
for individual, or-^ organizational purposes, Seve ial organi2ational and. 
personal motivation factors in this dimension are discussed, ffh^" 
third dimensioli, the evaluation structure, ent ails the ext-ent to 
which those commissioning the evaluat icr "^ha ve a specific set of 
'expectations a s to the areas to be investigated, the goals to be 
examin'ed, the role cf the ^valuator. The three diBensiens y 

comprise a^^thod for providing an ewvaluaticn consistent fcith the 
contextual needs /of a cjiveji situation., (fiutfiro/JAC) 



\ 



r 



> Re prod'uc tiols supplied bV EDBS are the best that cer be made * 

« * from the or iqi ca 1 dccume E t . * 

. ^ 3* ««««« 4 « 



EKLC 



^ Centec for jhe Study of Evaluation 



LA 



% — I 

CD 
UJ 



^HCLA^-Qts^uate School of Education 
, Lt3S^^igeles, California 90024': . 



U S OCPARTMCNTOFMCALTM, # 
COUCATION ♦WELFARE 
NATIONAL INSTJTUTE OF 
EDUCATION 

THIS DOCUMENT HAS ^BEEN REPRO- 
' DU^ED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORIGIN- 
ATrNGlT POINTSOF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRE- 
' SENTOFFICIAL NATIONAL INSTITUTEOF 
, €DUrATlON.POSa-lON OR POLICY 



■TERjyilSSION TO REPRODUCE THIS 
MATERIAL HAS Bt'EN 'GRANTED BY 



TO THE EDUCATIONAL. RESOURCES 
INFORMATION CENTER (ERl'cf^AND 
USERS OF THE ERIC SYSTEM." 



B. 




m , 

rmm 



Si. 



Center for the Study of Evaluation 



The Center for the Study of Evaluation (CSE) is a university-based 
research and development center established i)iM966 by the De- 
partment of Health, Education, and Welfare./TTie mission of the 
Center has been to conduct research, development, training, and 
dissemination activities for the purpose of improving the planning 
and evaluation of education and other social action programs. 
More specifically, CSE activities include: guidance for. educators 
and researchers in the areas Of tests and measures and in educa- 
tional decision making; development of easy to use evaluation pro- 
cedures and- materials m the form of manuajs, workshops and kits; 
and evaluation of projects for federal, state, local, and uriiversity 
agencies requesting technical assistance in evaluation. The work 
of CSE during its first decade has clarified the ways in which sys- 
tematic evaluation can be helpful in the improvement ' of many 
kinds gf programs. 
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/THE TITLE OF THIS SPEECH IS ... - 



^ ' ' ^ May'vin C Alkin 

The title ot this speech is: <?(choose one) ' ^ - « 

'.\ ' • a) Why Evaluations Don*t Work. 

^ ' Confessions of an Ev^aluation Sinn.er 

Testimonial of a TwiceTborn Evaluator 
(J) A Description bf the 3^ Dimensions of a 
> ' Context-oriented Eval u>tion..Ma'trix 

e) Musing and Mutterings of a Middle-ac/ed 
( Moderately -Middle-aged ) Evaluator 



I usecj to think many things that I no longer" believe. I believed th-ings 
about niy physical abilities^ T fc)elieved things about the extent to which I 
could drivel myself to further achievements. - I believed things about the 

L • ■ ' ^ ■■■■■ - ' ■ 

extent to which cert^iin kinds of accomplishments were important to me, 

. i ' ' ' ' ' 

Perhaps even I believed things abou>^ my sexual magnetism. And{ there were 

other beliefs. Many of them also haye been dispelled, partially as a function 

(I expect) af the aging process. 

. SoTTie of; the things that I believed about evaluation have also beerr ^ 

I 

modi^^S^d. Perhaps ten years ago, when I left my other disciplinary endeavors. 

' . ^ i 

in £ducationl and became "born""as an evaluator,- I surveyed the fieTd, noted 



^Keynote speech presented at the annual meeting of the California 
Educ'ational R(^search Association, Burlingame CA,. November 18, 1976. 



•the extent, to which there wer'G so marly , mi sgukled practitioners, and decided 
that knowledge by me gf the "true" v/ay of eval uati on* and ''personal practice' ' 
of the "rigfit" way of evaltlation were not in and of themselves sufficient. 
Indeed, what was necessary was a massive missionar'y effort at cgnverting the 
heathens (and assorted other misguided individuals) to what I ca^lled the true 
rel-ig-ibn, "decision-oriented evaluation," ' , 

I felt then that there was clearly a need to do something about miscon- 
ceptions of eval uatipn.v ^-s-Jooked around, and saw the host of misguided 
evaluators. ' . ' . " 

Look a't them, look at them, ^ thought I, glancing at the host of researchers 
pretending to be evaluato'rs. Don't they know, don't they know €hat research 
is not eval i^ati on ? - ' • , 

Webster says (dictionary Webster) that research is "studious 
inquiry^, usually a critical and exhaustive investigation or experimentation^, 
having for its aim the revtsion of accepted, concl usi b'ns ^i n the light of newly 
discovered fapts." And, oh how I hammer^ at that. Experimentation? Was 
evaluation experimentation? Was the aimWf evaluation simply the revision of 
accepted conclusions? Certainly , eval uati otk had as its goal something other 
than adding to the body of.scientific knov;}{edge. Surely evaluation had as 
its aim sometliing other than di scoveri ng-^addi tj onal insights into the nature 



of an entity based upon^ incontrovertible evidence. Clearly evaluation is 
diffore/it from research. Clearly eval uati on meant more than (or\less than-- 
dependiVig upon your bias) the accumulation of research findings. ' 

But I felt thai this was understood. The distinction .between evaluation 

and research w.v. fairly broadly accepted. The message was that research con- 



ditions are not-^ways possibleP^nd that research conditions may 



nc 



most appropriate in the conduct of evaluations in real world program settings. 
Schools are not laboratories; cliildren are not easily experimented with; and 
the demand for rigorous experimental control frequently is not possible in 
educatiorfal administrative settings. This, however, did not mean that the 
evaluatpr should not attempt to approximate research conditions in the field 
to be the best extent possible . But, research conditions could usually riot be 
met within school contexts. Moreover, the concern of the evaluator as opposed 
to that of the -researcher was the provision of information, the best possible 
information under the conditions existing within the field .setting. 

And so the gospel got spread. But, there remained some disbelievers. 
There remained some who stilKfelt that good evaluation was good research and 
that bad research was, of necessity, bad eval uation— an evaluation could only 
be clpissified as "well done" if it was good research and anything thought of 
as a poor research study mus,t gu-id -pro quo be poor evaluation. 

Also, there were the Jiieasurement speci al i s ts- and th§re wet^e the 
stati sticicins . . 

' Look at them, look at thefm, look at them, t^0ught I, glancing at the host 



of measurement specialists pretending to be evaluators. Don't they know, 
don't/ they know, that evaluation iswmore than Simply measurement? And' so, 
the tradi tional measurement ^masters , the measurement, establ ishment if you 
w.ill, wisely shook their heads; recognized a- rising trend, saw this new word 
"evaluation" coming over" the hori z6n', 'S^nd wi th perspi caci ty and sagaci ousness 
sutcumbed to the new movement.^ ^^nd how did these traditional masters of \- 
surement-- the nnmi-referenced test maker^s succumb'? iimple, they continued 
writing their measurement texts in precisely the same way, with the chapter 
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headings almost exactly the same, with the content almost' identical., but 
changed the titles of the books to "Measurei'iient and- Bjafuation in the Schools'"-, 

"Measurement and Evaluation in Edxicati,on and Psychology"., "Meastirement and _ 

' •> ' 

Evaluation in Teaching". (And these.,are, as you know, honest to Gosh titles.) 

^ ' - ■ . - ■ 

But, the emphasis was the same. The claimed synbn.ymity of- the words ' 
v ■ , ' . -/ 

"measureTne'nt" and ''evaluation" was, implied in the very^itle of the books. 

' And, wrten they infrequently ventured into the real world to do aij^.eval uation,- 

it was simply a question of selecting or devising, a meawe and reporting 

the results. That w-as it. .Evalua'tion was measurement of results. -PERIOD. 

• But there was, another failing of the work of the measurement masters— 

the results. And^hat results! Results based al=most exclusively on the use 

of norm referenced tests whose match with program objectives was dubious at 

' ; best. ' ■ ■ • ' . ^ 

But ttiere were some, there were some among the measurement special i^ , 

■who saw the folly of^ome of tKis reasoning. There were some among the modern 

measurement sages, who recogn-ized the error of basing evaluative judgmepts 

iQ- school situations' upon the results of norm referenced tests that m^^t or 

might not'(usually not) have a relationship to the content taugl|t in educa- 

. ational programs. This cul t devel oped holy werds that all throughout the land 

cam'e to know - "Behavioral Objective,," "Measureable Objective," "Cr^iterron- 

^'Vrefikl^enced testing," "Objectives ijased Evaluation." -The dogma was learned; tpy 

.fall, (no, peVhaps'experienced by alT'-is a better way .f'puttiihg i,t). "iho^_ 

Shalt wriLf i;ehdvioral objec ti|^es ," "Thou shalt write them and write them and 

write them." "{hou sh,a 1 1 write tjiem in measureable terms," "thou shalt write 
\, ■ ■ . . 

Lhem upon thr doorposts of Thy house and upon'Thy A127's," moreover, "Thou 

\ ■ . ^ . 

shdlt place them in column F on page 6.1," 'and so on. The;expenence was 

\ ■ ■ ' - " ' ■ 
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religious. Small^doses of 'dogma, with" similar small doses of practice led 
to"^l^??^ings|of se^ f-righteQusne^ss^. Large d§S|^^^ both led to feelings of ^ 
indignation toward non-believers. .. ] • 

Yet, the approach remajned siniplistic. The key to evaluation was 
. measurement' . True, a new' 1<j[nd of measurement, but measurement and evaluation 
jwere still taken as synonymous tenns".'"W6 measure (properly measure. in cri-, 

h ' " ^ ' -'"^ - • 

terion reference terms that is), to measure was in essence^ to! perform' eval ua- 
t^ion. 

But again I implored, don.*t you see that evaluation is far more than 
jusTt measurement, don*t you .seg- that^'the decisions to b^^ made as a consequence 
^f"Kthe evaluation perhaps have as much to do with«the way in which the eval.ua-^. 
"tipn'should be conducted, as do 'the objecti^s to be measured. Measurement is- 
a part (P^grant you\' an important part) but nevertheless only a small part of 
the science (perhaps in rea-'l ity/=i t is an art) af evaluation. 

'^o no avail. M^yiy^eard anfl marjy heeded the call to decision-oriented 
evaluation, but the seductive entrappings of this neatly presented, enter-^ 
tainingi and simple road to-eval)[i(ati on salvation gained many followers. Didn;'t 



they know, d\^dn't they know, that mfeasurement- is just orie part of evaluation? 

But the mistaken measurement specialists were not alone in th-eir innocence 
^ there were alsjD the ^sophisticated statisticians. . 

Look a-t them, look at th,em, thought-I, glancijig at -the host o1^ statis- ' 
ticians acting as though statistical methodology , was synonymous with evalua- 
tion. At that, most of the stuff* that they were^^^pig^l ing was psychological . 
statistics, the statistics of psychological experi.mentati oi:^^ What about 
social Research methodology which .in many respects is more appropriate for 



the kindi of si tuatiofis faced by most evaluators? ' vlhat about the ^s^bati s ti cat 
procedure? of the economists, as they cast light upon cost effectiveness kinds 
of questi^s' ill evaluation?'^What about th>} sta ti s^'ti cal procedures -^empl oyed in 



operations research nia.de_l-S?^ • . o 

Surely, s ta (^^4 tiical analysis is an inijiortant part of the evaluation 



operation. ^But® hear them talk, one might tjiink'tbdt the majorVfai^i ng of 
the Coleman Evaluation was' the wayHn whi^h the 'analysis was done. Article 
after article has been wrl^tten criticizing the analytic procedure a^d sugge^^^tuig^ 
even more esotericj^iodel s for "fine tuning" the data. From this point of vi^w, 
the faf-ilings of the Colejnan Evaluation are not related to the appropriatenes's 



the data col 1 ected , (to the way it was collected-, to the attention (or lack 




of attention) by .'the evaluators to the .pol i ti cal context, or to\he attention 
(or lack of attejition) by the evaluators to other^^xternal data sources ^that 
mi^ht be viewed arid considered by those'^ji major policy positions. All are 
.incidental to the mSjes^bi/ of the an^ysls. The sophistication of the statis^-r 
' tical sophistry reignl supreme. ^ . _ . ^ 

My words, and the words of other similarly inclined evaluation prophets, 
fall on deaf ears. For many, evaluation was synonymisijus with stajiistica! 
anaVysis. ^Wt^ there v;ere a few other things that one 'thought about, J;)ut it 
was- pr>imari 1y statistical analysis, Wouldn't they accept, coul dVi ' t j:hey accept, 
that statistics is.just one part of evaluationT^ " 

In ^juxtaposition to these feelings of what evaluation was not, I ar}d 
others had be^n pulling together thoughts on what evaluation is., t had 

- • ■ ■ ^ ' . ■ ■ y ^ ^ 

been at U.C.L.A. developing, refining, and aging my brand of evaluation stuff. 



btf)t.-rs .h.dd btjcp tillin^j a si mil at, but di^. t.dfit, (^valuation vine-yard. ^ 

^. ^ 



/ 



Dan Stuff lebeam,. and the late Mai Provus, tO' pame two, had each been » ^ 

approaching evaluation in a s'Knilar way\ (Indeed ther*e were^soine who scof- 
fingT^dubbed us fellow "handmaidensy of ^ecision makers"--referri ng of cour'Se' 
to 'OUT concern for .r^ognizi ng' the speci f 1 c deci sipn issues involved, ["Hand- 
maiden" thought of Dan Stufflebean) wearing a scanty swimsui t , ^ rose in teeth, 

laurel-^Qn (head, carrying bowl of fruit, tripping lightly '*(?) through woodi , 

'\- \ . ■ • ? ^ ■ ■ 

following a^ecisionr maker]). - '"^ ^ ' 

We a11 said similar things about what evaluation ts not > Evaluation is 
,not synonymous with researchr :Measuremen-1r alone is not the essence of.eval- 
uation. Statistical analysis alone is not the essence of ^eva^ uati on . 

We all believed that the purpose of evaluation was the jcol lection, anaiy^ 
sis, and reporting of information relevant for specific decision concerns , 
i Evaluation must^be decision-oriented and the activities that the evaluator 
,;;[^j^ngages ;*n must contt^au.al ly us^*^the decision cc^cerns as a gui(Jing framework. 

We all defined various deci si on l&veTs" or deci si on .types that we felt ; 
occurred as a part of \the edjjcati onal decisioff process . The distinctions 
between the decision types that each of "us identified werje based 'primari ly 
upon differences in point of view, and differences in the kinds of school 
-systems that we had been exposed to,, or were operating in. 'MaK Provus, for 
example, was at the. time serving as director of research and ^val uation for 
the Pittsburgh Public Schools and had. to be particularly tuned to the needs 
and^organi zati onal functioning of a district such as that. My^own experience 

4 

/ Jand orientation , .perhap^-t was towards the middle-sized kind of school district. 
He ^11 tried to develop specific evaluation proc^ures that could be 
employed ^y evaluators in acquiring the kinds of Information necessary for each 
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of the decision types tPiat.ha^ been stipulate d... In 6ur own ways, eacPi of us^; 
attempted to clarify and define precisely the steps that a decision-oriented 
evaluator must^ follow in order co be maximally effective. - -In , our own ways, /' 



each of us at tempted"^ |:b disseminate these views on evaluation , procedures to 
practitioners. I, for' rny pai^t, [.ps^'ructed efforts at dissemination of eval- 
uation information, tlu'ough tlie ^Center for the Study of Bvaluatj on which I 



directed. I beg^n the process by. creating a variety of evaluation training 

materials Evaluation Workshops/Ne'eds Assessment Kit/Other Evaluattjon' Kits 

for^Ll^e^ntary School Practitioners.' ^ V > '\ ^ . 

J I believe that the deci si on-ori ente^l view^f evaluaiton enjoyed '(and , 

enjoys) some modest success. Many havp^^adopted the viewpoint that evaluation 

has as its-i^rime function the provision of information for decision making 

^ (that th,e -^evaluator has, as^ one^- of his major concerns^ to deterj^ne the' likely^ 

decisions prior to commencing e'valuation dati coll^ctjon or analysis/. 

And nov/i' for the confession that I had. earlier pitofinsed. 'Tr^c-epts that " ^ 

*I have preached are n^t wi th.out\th^i r failings.*'" (Let's see, it says here^. 

[looking down at notes^ "Leave plenty -of time for the gasp of astc^ishment 

from the audience to subside before continuing"). Yes, many peoole -.haVe -seen 

..• . 

J * • ' i^. r ' 

the decision-oriented evaluation point erf view., many people have seen tpe ' . 

• r - ^ : ■ ■ ■ • 

li*/ht.^ Perhaps the problem is that some have seen too much ligh^* . 

• ^ The naivete and the si mpl i ci t^. that I abhorred i-n other approaches is 

equally prese,nt in^the practice, of deci si on-oftented evaluation -a^s conducted 

by many. These deficiencies stem primarily from the incorrect assumption by 

some that the evaluation > decision-making link is clean , f i rm , direct and 

wiiiiie(j>ate . Thr. d^suinpt i on invariably leads:^ to the understanding that the 
— ^ . ' ♦ ft. » 

-/ ■ .i 



* ft' 



evaluator is lo (leteniiino wlidl doc: i , ions drv. likoly,. arui havirujMiiade tf>^se 
deteniiinatiOns , he/slie conducts fiis/hof^ L?vaiiiaLiofi and presoni.s- findinys, 
th^irn sl'.^ back'aiid r^iijoys U-a lauiis f'f his/lier afrcrr,- The a-^ i lai is- 

l.no t: ■ ava ! ua t: .. , aua-./, \>u \ i):'; raa^Mj^erl, baMcva_^[, ^V?'^^, '^''^ 

iaiis dj.^a : aa .i aj ^'J^a^o,■■ i iy ' /a-^i -hai he's, aaa; '^aei ved^and 

iip;) 1 ,afiia 1 Oil . 1.^! Jh : . ..(.lali (;a Lha^;. va 1 i.'vL i i ntaa ' • ' • and 

a--, i 1 ai ! ; :r. ■ ' ;i iiii-r.i; u;a)jt ::laais"aa iiu'.: ]a\ 

iiifi^;' ^ ,i ijck hiipaoi. (ar ava I u^i luai aii a; ^ ■■.h... ... M.-.a;!i:>j 

ia:lifi-.j La ;c\a a i Liiat ihara a'lV other .ini-^ais' iji..a '.ru, ujaj;,n)n !aa.a..;a^!> , ar 



! iaj'i e Vfi 1 I. i an 



faaai O;- " ] ^ a a " a:, ; ■.:e(:l -.a/j ! a. 



• . i aa . ah^M^'^ ^^h ' >. i i ' = h-: . I l m: i ; fir -ssa,ii;)i. ] • M'^. 

•: Lna aa 'J". ' -.a/: /■.a.j , i; ^a^aa- aa '■ [yj a-iia ".hoi'e oaaiV, i)- oi-.. 

[,..,,-:■, {)'•.! ai d:: f i !. i L^ata i.-:-. .-^ iia;iii) a^ oi inoni.ia, a^I'^ whtMt 

,.^;,iMt La d^•^Vi(■;■ .■ '-.a.^.-;!; an a v.i i a .\ wa> ..a^a a.^^^ .a ^ajCaiiU: '^:a: of ^ 

. \: PA « .-a.M' • , - - !a ! • v-'Vi-^ /•ai ua U . I i;.a .iin- , as: ; ; : i . Jiat 



o 1 i.ru: a.;-; , aa : 1 . ■ (J ; i ■ aa. LfU' 



. 1 u.t ■ 



I a- 



i u ; ■ 1 aa 



o r(M a- (" 



fi .i i M( j \ "I 1. 7 ' . ' ; ; . . 1 1 r; j I fa- ( !* ' ! ■ . i ' at l.a \ 



! i : .ii) i c:li 



■ i:. • H-' w'a a-l-jv-a .It-iiMtwi lii.iiia',, ! ^ 1 kal .t[)Liul ilia !• i na'> oi 
j, -MM'- I iV 7 a.i ! . |f;-':t to {i[rrM\a i nos'.: '"ia^Mt!. Into (Iim i , i - i <jiio<; 



iiuna .-r-* Ih'^; l;-.(a',a'a taia^/ir--^ n{ nji,.t pr^'^an (a 1. 1 on Uia t. vaju i <l a 

. ' / . ■ ^ a' ' > 



■ ■ ■)• .. \ 

,> considered fey the decision maker "as an adequate response to the decision ques- 
tion that he had in mind. Thus, in essence,-! as the evaluator simulated some 

evaluation-^resylts ("suppose I to.ld you that X percent of the students had 
' accomprished such and such an objective " ; "s.uppose I told you. that 75?;', of 

the parents v:i th- chi 1 dren in the program avernged a scori; of 4.5 on an attitude 
tov/ard-program questionnaire v/hr o 4 is favorable and 5 signifies highly 
fdvorable"), dnU in cho process of this refining the conditions of adequate 
evaluation inronnatK i. decision males r and evaluator gained further refinement 
■ of th,o decision question. Typically this refinement would occur as a conse- 
quence or a decision maker indicating that the data presented would not really 
be adequate for him co make the kind of decision that he wanted and in fact 
•Si^^iiidybe ijie way in which the decision question had been stated was in- 
^curale. 'And so, tfie evaluator- and decision maker would jointly redefine 
decision questions into a form that more properly reflected what the decision 
i,,ik;!r-or <-■;.;'<: i'.iori makers ) .-real ly nieant. ^ , ■ - 

i ..,):, ietcu' ihaL cassette-tape, i.ii.ied "Framing tfie Decision Context 

■-,:r. :^oir -.c ;cv;iuri: uneasy about che i:ape. I fsU uneasy not because of the 
,,u,Hu,y of the i.apc, (wbicri of course 1 wa >Mvinced was "great") but' rather 
the uneasiness s lenmiod from the fact that part of the discussion of the tape 
.Icait wifii Mfuafiod', fiiaf mi()ht .logically be ca 1 1 ed' f rami ng the nori-deci si on 
(^,nt',!,;i . ih.iL IS, fru' ev.iluafor in flie ^course of the "framing" mfght deter-^ 
„ii,„- tlidf indcfMl HI -many instan>-.-s there. was no decision that wds to be made 

,1 T-<-,uil ttiJ (.■valuation i n 1 o rma f i on ,' (;r that tiie likelifiood of decisions 
w<i'. r(ifl'ioL". l^M7v<)[)S <\n()l\\L'r^ nuwnu for i)n(?as i ncoT. vyas Cluil I wao aware of 
i ,|^-p.,iL n\ non-Mi. M;i>iOn rol.UxMl (-/.i 1 iia L i oK arl.ivity UuiL pcrliaps was not 

Mi-.(,u'/.(.(i wii.fiin lh(.,' Al. KA (,ass(M, t.(-- {.a|)0 . 

/ 10 ^ 
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iEvaluation does not always le^d to decisions that will be made. 
Occasionally, possibly^ even frequently, evaluations were never intended to 
lead' to decisions. Thus, what I was driving at (what' K was concerned about 
/at the that I devGlo(3ed that tape speech), was perhaps not the decision 

context but the necessity for understandi ny the total context in vihich the 
evaluation- would be occurring. v ' 

- University professors are of Lbn. s cereotyped as ready to develop a model, 
a theory, a foniiu i a ci on , a frainework, a conceptualization, a tautologocjy, a 
theoretical ?omiulation, or any uf dozens of otiier esoteric conceptual enti- 
ties, at the drop of a- hat or at tli^ merest i ndi cati on ' of the potential 
possibility that 0!ie miqht be helpful.' I would find it helpfu 1 to develop a 
categorfzation (you'll note chat that was not one of the terms I previously 
ridiculed) showing three dimensions or way§^ of classifying the evaluation 
context; tl^ere are undoubtedly others that might com-^ to your attention or 
which might boicmi^ IiihikmI i ale ly apparent^ for rnoment I fir)d tiris a iK^lpful 
way to thml: about the evaluation context. (I can't really say how long it 
wi 1 1 remain unchanged) . 

The ru^sL difnensidn of the evaluation context is something that J. will 
refer to as eva]_uji^t_i02i Jii_t^^ As I look at the research that I had done on 
evaluation utilization, at the case studies that I am currently conducting re- 
lated to this par^ticular^ topic and Lo the evaluation projects that I liave 
personally condu/:ted in s-clu)ol districts and in other kinds of organizations, 
I l)(M.ome aciitiny awc^re of fho necessity of nhdung thi? distinction between 

evcHuations which ^\vo commi ss i orifMl for tlie [)ur[JOse of obtaining some yx^sults 
\ 

[or learning al)0(il soiiip outcomes ol ttu? |)rogram) as opposed tp those evdlua- 

t 

tions in which tho major intent is the commissioning of tfie evaluatior) itself. 

\ - • 
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In the latter instance, it is the proces s pf having the evaluation conducted 
that i$ important, rather than the results that might be obtained from the - 
evaluatioru In some respects^ (he commissioners of the evaluation see it as 
almost immaterial whetiier or not the evaluation uUimatel^ produces one set 
of ^findings or' (Uiotfier' (or none at all). 

As I've indicated, one [)oteriLial evaluation i nten l. thoii is engaging in 
' the '[)rx)cess . - Obviously another possible dimension of evaluation intent is 
hdvmg d concern tor the results or an understanding of the outcomes of the 
progranK But 1 v/ould like to break this into tv;o categories, the first, a concern 
tor the results witfi d pdrticu I ar predispositi on as to what the nature of ^ 

those results sfiould be, and the second, having a concern for the results 

/ 

<i?nere there i du ( jpefiiin ndedn ess on the part of those commissioning the 

evdj ua ti on . ^ 

I have become awar^e or instances in which those' comini ss i oni ng an evalua- 
Lifjh rv^,Uly lt<iVL' dotie so in (y^dt^rr 10 "show that the program' is doing a good 
I'jb." ' Ttie intLMit is nci to " do Lcniii r!e ,U tffe program is do.ing a good job" 
but to show. .Hopefully, instances of this type are few and lessening. Hope- 
fully, mor^e and niore evriluatoi'S are refusing to participate in evaluations 
whiLh fidv(.^ this ns 'thu intent. 

'irius, 1 set: tfirec" (dtogorn(^^ of evafudtion intenl: 1) .process, 2.) r^es'ults 

I 

wifdi J [irrMli sp()'> i 1 1 on and 0 results (or results openninded, if you wish). \, 

i^ow lot ino ..OfisidfM' d second dimension of the evaluation context. (Help 
irj< ,jrM,ido do uAwwrlor idiis dimension of ttie evaluation context.) It seems 
t.(, nin i\u\\. ('ynliidtHHr. arc commissioned either for^ individual or for or gani - 
' Zcit.Kjricii purposes, and iiir]r(M)vcr^, tliat t[ic vdr- i ty o f . organ i /a ti ona ) i)urposes 




differ raither substantially. For example, I can cuiiceive of an Instance - 
v^here an.igval uati on might be tommi ssi oned'^ to fulfill personal needs of the 
individual requesting the evaTuation. That is, the function is not so,, much 
the demonstration of a program but rather it is an action that is taking 

place and v/hicr: haS been ccnmii ss i oned by a decision maker, primarily because 
of a view thr^t it might aidd to his own personal glory, or to his status 
within the district, or to his feelings about himself, or to the way his 
colleagues in other dis tricts v iew him . 

I suppose there is a cerLain amount of the personal element in every 
situation. When 1 describe the alternatives within this dimension, I 
suspect that they are not mutually exclusive, but. that rather we're concelr:ned 
with the extent to which one or another of them is the dominating motivation. 
As a second al ternati ve witfiin this dimension, I suppose that one might 
think Lhat the dominating moLivaLion could be found within the school program 
i tse n\^(pcrhaps there is an insisLoiice by teachers or by a iv.'ogram director 
that an evaluatio.n be conducted to find out how "we" are doing Possibly, 
the dominating motivation is at the school or district level. Possibly, the 
prime mover is the principal of the school or the Superintendent of Schools - 
although my experience tells me that this is not usual 1 y^v-^he case because 
they typically do not want to shake up the system unless it is required from 
G 1 sewfiere . 

A furth(M^ possii)le motivating force is frequently the school board. I 
reLall performing an (?vdluation for a school district of arjj- alternat i ve hi^h 
scfu)ol. In that Las-G, i L was tUi) schoo 1 ^ board 'that insisted that an evalua- 
ti/on take place. (Something novel was being introduced into a relatively 



con'seffvati ve school ^ and they wanted the a of knowing whether 

It worked — and *if it" clvdn ' t, they wanted the political i/nsulation of having 
called foY -the^ evaluation that demonstrated it.) Another possibility is that^ 

the dominating force, the motivating force, might come fi'-om the community it- \ 

- I . 
self. Perhaps dn,.exdmple of this might be on hig^~ly volatile issues v^/hvl<:h ^ 

invoke a great deal of cominunity fervor (e.g. busing) and an evaluation, 
(generically^ sofiie kind of data acquisition and summary activities-) -- an 
evaluation is called for in response to an evaluation motivating force from the 
community. ■ . 



A final , example of tlii s dimension of the,->„^ntext is theexternal agency. 
And- I feel that this is a parti cularly appropriate example beca^use many 
-valuations, (possibly most evaluations), are commissioned primarily because 
of an e)^ternal agency (a state or federal gove.rnment) and- the requirements 
t:fiat agency has placed. upon organizations that receive program fut^ds. Let me 
cake this one exainple ^and explore it somewhat further^si nee >.i t Jorms such 
a major por.tjon of what,- is done^-under the label "evaluation.*' 

Gg^)^iiment agen/cies impose reporting requirements on school districts. 
These^T^eporting' requi rements demand the presentation of certain kinds of data 
someSS.'wfti^'h. is outconS data. .These reporting requirements are generally 

' ■ 

y^AW&^'^l^^'^'^^ ^i^td^^^^^' by the agencies. 'd "evaluation" i| in 

The pr'p.^^^;wh^^p^}/if<|^.we re felt "i.. ' ■.-),• Lant, evaluation |i;tivitiies 



that W%fli b(^&^r|^^pi^3,cal-ly, procedures get modi'fied, re'nined^ 



enlarged, reifVef i ne"d7.'>e-mad'i fi ed , re-^enl arged . And while the intent (usually 



t>'?,%.g3.vrde an i nfo'^iati on 'Sys tern and an evaluative procedure useful for 
^rJec'i'i.ion makimi at local levels and at state levels) may have been cle/ar 
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to those who init'ially conceived of the system, each subsec^ent refinement 
to the procedure makes the rationale less clear. Soon, thos^/who become 
enmeshed in tihS procedural contortions become a captive (^f the procedures - 
and in a very real sense, the agency is the dominating force. Domination 
by the agency usual lv>^ precludes whatever valid deci si on. uti 1 1 ty interft that 
might have initial^ bee^n present as the cornerotone of the "'eva'lua- 
tion procedure." In practice, agency dominated or motiva-^d Evaluation 
frequently has little if any impact on decision making,. It W.blild be sur- 
prising if thi,s kind^f evaluation did have impact. The sma■>les^t amount of 
knov^vledge of how federal agencies end other large governmental bureaucra^es 
operate quickly convinces one that this is not an aren^^ in whi ch empi rical 
rationality i^ always the dom|nant mode. Furthermore ^ 'a little common 
sen^e, undersHraTi'-di ng of the time schedule wi-th\which funding decisions get 
made in atjenccies at the state and -federal l(evel (for example), in juxtaposi- 
tion with tht^ tiiiie of the year in which governmentally Required "evaluation 
reports*' are received, quickly cf^)nvince one "l^hat there is no way that such 
reports ca>n haveMmpact upon fundi ng.^deci si ons . Agency-required evaluation 
reports ty'p'i'^ally are received from three to six months a • the fufiding 
decision has already been made. 

Typical ly, tHo only way that the eva1 uator can have impact in such pro- 
grams, in teniis of the provision of evaluation information leading to program 
mod,i fications, IS for the evaluator to^ develop a relati onshi p wi th project 

personnel? where project personnel are attentive to the evaluation findings, 

- - ■ . • 

respect the evaluator and his i n tecj ci by , ' and are aware^of major evaluation 

.> ^ 
findings, g/ior to the publication of the final evaiua.tion report, and are 
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able and willing to incorporate these recommendations into progranl^changes _ 
for subsequent years. In many cases, the routinizati\)n of the procedures 
convinces tjjle evaluator that compliance with agency requirements is synony- ' 
mous.'with good evaluation and doing more is redundant. 

(•(This belief -is rei'nforced by many practicing school, administrators, • 
who believe "that evaluation is simply an event that lelds to compli-^ce with 
various WnrTrequirements. There is no real expectation that major basic 
decisions will be made. The name of the balOgame is simply not to get 
"dinged" by the governmental agency -- in- short to show the agency that 
you've played the g|^e their way.) ^ >^ \ 

Let me summarize one point with respect to this second dimension, and 
.that is that we are not concerned with who actually commissions the evalua- 
tion, but rather the question is who or what is th§ motivating force for 
the evaluation bei ng^commi ssioned. In the past, when I have placed emphasis 
and concern upon designation of the evaluation report commiss-ion- - ' 

son who actually set in motion the selection of an eva ! 
i,, ■ - - . 

me that ne ri^t question in cemis of uride ing the 

(L . ■ 

relationship between the evaluator and the person he was to r... /rt to but 

/^'^ . > ^ ■ ' 

itimissed a very important dimension, and that dimension dealt with who or 

/ A 

what was th'e fiioving force that necessitated an -^vaTuation . Now what name 
would you give to this second dimension of the evaluation context? (Shall 
we call it the eval uati on motivating force dimension? Kind of clumsy -- 
perhaps you'll help me think of a better name for this dimension.) 

There is a third dimension to the context of evaluation which I would 
• like to identipfy. This dimension deals with the extent to which those 
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commissioning the evaluation have a specific set of expectations 3s to the 
areas to be investigated, the goals to be examined, or what it is the eval- 
uator is to do. Perhaps one could think of the ends of a continuum as 
restricted agenda evaluation or open agenda evaluation. It seems to ine * 
that:^this description goes/ far beyond the goal-based/goal-free distinction 
that Michael Scriven has made in his writings, because we are dealing with 
far more than^ the^^goals of the organization. Of concern nfight alsa be 
program char^£teris tiks and the extent to which they're implemented. Also c 
included wXi;iin this category is ^general expression of the auton^y granted 
to the evaluator, pointing at the program to be evaluated and asking him to 
present a report. The typical program evajuation situation of which most of 
us are .aware has a tightly prescribed^'agenda within which the evaluat^ - 
expected to live. Typi cal ly , -this tight pres on is related to the 

,or completing a report to an external agency. However, even 
withouc such a report to an external agency, I have performed evaluations in 
some instances where school districts have let out requests for proposals 

V 

that specifically state the activities that the evaluator is to engage in 
and the kinds of data that he is tQ collect. (And I am sure that many of you 

a 

have performed such evaluations as well.) 

At the other end of the continuum, I can recall somfe' instances of eval- 
uation situations with "open agendas." In one case, I had been requested by 
a State LegisTative Conmiittee to "evaluate" the educational program of 
that State's juvenile detention facilitfes. This evaluation was to take 
plaVe within. a three da^ period. And there were little if any°addi'tional 
prescriptions presented tcf me on the way in which this evaluation would take 
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place, the data v;ould be collected', the way jn which my observations woiild 
take place, or the way in which my final report would ultimately be framed. 
I- was the evaluator. I had perfonned evaluations for this State" Legj slature 
on a number of other occasions. The legislature had been impressed "Hnd 
pleased with my work. Thus, in tf>is instance, there seemed- to be a 
belief in the evaluator as an-omnicient presence who needed no guidance or 
framework, who was neither to be bounded by the goals of the program, by ^ 
reporting r^equi r^ments and fonnats, or by specific decision concerns of 
those responsible for programs. In this instance, what was wanted was->a 
sage, a seer, perhaps an oracle of Delphi. (T hope ihiil when s^uch oraches 
are commissioned, that those who do so get not the oracle but the true 
priests, [^tory of DELPHI ~- stone, smoke rising, etc., a mad illiterate 
woman, priests who interpr^eted her ramblings into meaningful" statements 
in line with the context.]) 

At any rate, what I have identified here is a third dimension of the con- 

4 if 
text of evaluation. Perhaps I might call this third dimension of tht evalua- 
tion context something like evaluation structure implying^the structure 



provided to the evaluation by those_>yho co^^ission it rather than the structure 
provided to the evaluation by the evaluator in the process of his conducting it 

What it appears tu iiie that I have said thus far is that the notion of 
decision-oriented evaluation (or of any other si ngl e kind of approach to eval- 
ualion) is extremely limiting and that instead we ought to be thinking of the 
context in which the evaluation takes place context-oriented evaluation if 
^ou will. And that one way of framing-that context , - one way of describing 
the context in which evaluation tai^es place is in tems of three dimensions 
that I have referred to as eval uati on intent, where the primary difference 
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is beljween wanting the eyal uati on ♦because you'\^^ant the results, as opposed 

to wanting the .evaluation simply to demonstrate that^the process has ^ 

taken place- The second dimension of this context^f eva1 uati on is the iden- - 

ti f^ation*' of the evaluation moti vati ng/^force the mo^i vating^indi vicJi^|^^ 

or force for the evaluation being cormi ssi'onedy'^ The third dimefision of the 

context of evaluation is the evaluation's structure which I defined as the 

- — f 

restri cti veness or openne'ss with which >the evaluation could be conducted 

that is the initial structure imposed upoftl^'he evaluation prior to the 

evaluator commenci ng- hi s work. 

, Now it appears to me that the first and major job^ of the evaluator, is 

to play to this dominant context, °to provide an evaluation consi stent with' 

the major contextual needs of a gi ven /si tuation o Or, alternat^'vely^'f that 

context is disagreeable^^repugnant, ethically unpure or what hav^ you, to 

either refuse to do the evaluation or (with greater risk) perform the eval- 

\ ^ 

uation in a manner not intended to serve thev-dbmi nant evaluation context! 

I ( 

But if the latter course is chosen 5 if the evaluator chooses to perrorm an ^ 
evaluation inconsistent wi tli the dominant context^ he ought to do it with a 
full understanding of the situation and the potential consequences instead ^ 
of with the naivet'e that currently exists in evaluators who sometimes per- 
fonn an evaluation, have high expectations for it and suffer the disappoint- 
ment of apathy on the part of those who receive the report. If the evaT^^^ator 
is aware of the dominant context and finds it appropriate to work within that 
context there is no reason why he should not be able^-to attempt, to extend the 
implications and findings of his work to other appropriate contexts or to^ 
feven attempt to change the dominant context. The important pointy however, 
is that if evaluation is to in any way be useful we need to be aware of what 
is the contextual starting point. , , ' » 
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Against these various contextual situations there are perjifeips, a' variety 
of evaluation f onus, and styles that become Imposed. D.ifferent evaluators 
have different ways of approaching the evaluation situation. Sometimes the 
mati:h with the context is. good, 'sometimes it is poor. When the mat&h of the 
^^valuator with the context is good, he will typically praise himself for an 
^£^aluation well done. But when the match is poor^ he will typically bemoan^ 
the lack of responsiveness of the various program p.artiCi pants to his evalua- 
tion: ' - . , V 

"Now, v;hat about, these^ di f ferent fonfis.^and styles? Evaluators approach 
their task /n di fferent ways. Some ^vaTuato^s have been labeled 'or Itave . 
labeled themselves as deci si on-ori ented. eval uators (they must use different^ 
models to. describe the steps in the sequence as^\^hey' see them) and in each 
situation they wlust struggle to identify the potential decisions even in 
si tuations^ V)^re none ex\st. Some evaluators prefer the role of seer and 
try to j^ffipose tijjat upon all situations. Some evaluators readi^^y accept 
the role of "federal form filler-outers" and "state statute satisfiers" and 
willingly do^^^^ i s necessary to satisfy those requirements. That too is 
evaluation. \. 

Conte^xt oriented evalijati6^i involves the necessity of recognizing con- 
textual needs and of matching evaluative orientation and style to contextual 
needs. To the extent that an evaluator has the insight and the abilities to 
modify hts style in his response to the ^contextual situation, he will be more 
successful. Generally, the evaluator cannot do very much more than the con- 



text will allow. Generally, evaluation wyll not be more than the context 

/ 

will allow, 
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It someone^^h-ould ask you (I can't imagine who.would)^ what th^t speech 
by Alkin was all about, it. seems to me that you might' tell them that Vt dealt 
with why evaluations don' tM^orl<^, that there was a certain amount of confes- 
Sim on his part as to how he felt he.had erred in his evaluation pres(^rip- 
tions of the past, there, were a lot of musings and mutterings.^ and substan- 
tively he focused or^ context-ofie^ited evaluati on the -notion that we ought 
to be aware of the co^ltext in which evaluation takes place and be responsive 



to that context, 
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